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The second six months of this grant saw further development of GT-CATS- 
the Georgia Tech Crew Activity Tracking System and progress on research 
exploring tutoring concepts for tutors for mode management. The latter 
included data analysis and a preliminary paper summarizing the 
development and evaluation of the VNAV Tutor. A follow-on to the VNAV 
Tutor is planned. Research in this direction will examine the use of 
OFMspert and GT-CATS to create an ‘intelligent’ tutor for mode 
management, a more extensive domain of application than only vertical 
navigation, and alternative pedagogy, such as substituting focused cases 
of reported mode management situations rather than lessons defined by 
full LOFT scenarios. 

The following sections further describe each of these areas. They include 
copies of conference papers and presentations. 



GT-CATS 


Overview 

The Georgia Tech Crew Activity Tracking System (GT-CATS) is a computer system 
which embodies a methodology and architecture for understanding how operators select 
and use modes of automation to control complex dynamic systems. GT-CATS has been 
implemented specifically to track the activities of “glass cockpit” pilots as they use modes 
of automation to fly a desired flight path. GT-CATS’ activity tracking methodology 
specifically involves: (1) hypothesizing the mode configuration the operator will select in 
the current operational setting; (2) confirming that actual operator actions support the 
hypothesized mode configuration; (3) determining whether a detected operator action in 
fact supports a valid alternative mode configuration, or whether the action is in error; and 
(4) identifying missed or late actions. The centerpiece of GT-CATS’ methodology is the 
capacity to revise an incorrect hypothesis about the expected mode configuration to arrive at 
accurate explanations for operator actions that support an alternative valid mode 
configuration. A block diagram of GT-CATS is depicted in Figure 1. 



Figure 1. GT-CATS Block Diagram 


Current Work 

Recent GT-CATS research has focused primarily on preparing for a formal evaluation 
of GT-CATS’ activity tracking capabilities. The planned evaluation seeks to demonstrate 








GT-CATS’ use and effectiveness for understanding how operators select and use modes of 
automation. The evaluation requires fifteen Boeing 757/767 type-rated pilots to verbalize 
the reasoning behind their autoflight system mode manipulations as they fly five flight 
scenarios on the Georgia Tech Electronic Flight Instrument Research Testbed (GT- 
EFIRT) simulator. The study will then compare the pilots’ explanations to the expectations 
and explanations issued by GT-CATS. Output from GT-CATS should agree closely with 
the pilots’ explanations, if GT-CATS’ activity tracking method is to prove successful. 

A considerable share of the evaluation preparations addressed the development of the 
experimental flight scenarios, to ensure adequate representiveness and realism. Flight 
scenarios from a major airline and a NASA Ames LOFT form the basis for the planned 
scenarios. The scenarios (Figures 2a and 2b) incorporate clearances designed to elicit a 
range of automation mode selections/transitions. Test runs have aimed toward ensuring 
that clearances are issued at appropriate times, and that selected automation modes 
perform in the required capacity (e.g., ensuring that the GT-EFIRT simulator observes a 
programmed crossing restriction in VNAV). As part of this process, consultations with 
actual Boeing 757/767 pilots helped to identify and correct problems with the GT-EFIRT 
simulator and scenarios. 

Along with the development and testing of the experimental setup, GT-CATS research 
efforts also addressed the development of a preliminary plan for analyzing the data 
obtained in the planned evaluation. Recent refinements to the plan enable more effective 
examination of GT-CATS’ activity tracking capabilities. The plan now allows for 
evaluation of the activity tracking methodology on an action-by-action basis, to assess in 
detail: (1) how well GT-CATS understands each type of pilot action, (2) which specific 
situations cause GT-CATS to revise explanations, and (3) which types of situations result 
in degraded understanding. Modifications to GT-CATS’ output now make this additional 
information available. The output data will be formatted with the aid of a special analysis 
program. The program is designed to tabulate, by action type, how each action is 
interpreted, and generate summary information that describes GT-CATS' performance at 
a level adequate for assessing its strengths and weaknesses. 

Presentations 

Much of this work was formally presented at the 1994 IEEE Conference on Systems, 
Man, and Cybernetics in San Antonio, Texas. A companion paper entitled “A 
Methodology for Understanding How Operators Select and Use Automation to Control 
Complex Dynamic Systems” appeared in the conference proceedings. The paper and 
presentation are included as Appendix A1 and A2 of this report. A presentation on GT- 



CATS was also given at the NASA Ames Training For Automation workshop (Appendix 
A3). Subsequent discussions explored possibilities for integrating GT-CATS into an 
intelligent tutoring system for advanced cockpit automation. 
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Figure 2a. Scenarios 1-3. 
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Scenario 4: KCLT-KATL1 
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Figure 2b, Scenarios 4-5. 
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APPENDIX A1 


GT-CATS SMC Paper 


Callantine, T. J., and Mitchell, C. M. (1994). A methodology for 
understanding how operators use modes of automation to control complex 
dynamic systems. Proceedings of the 1994 International Conference on 
Systems, Man, and Cybernetics, San Antonio, TX. 



APPENDIX A2 


GT-CATS SMC Presentation 

Presentation Entitled “A Methodology for Understanding How Operators 
Use Modes of Automation to Control Complex Dynamic Systems” presented 
at the 1994 International Conference on Systems, Man, and Cybernetics, 
San Antonio, TX, October, 1994. 



APPENDIX A3 


GT-CATS Training for Automation Workshop Presentation 

Presentation Entitled “The Georgia Tech Crew Activity Tracking System” 
presented at the NASA Ames Training for Automation Workshop, Moffett 
Field, CA, August, 1994. 



Training Systems for Automation 


One of the primary research efforts for this grant is training systems for automation. In the 
past six months this effort included several complementary activities: evaluation and 
documentation of the VNAV Tutor, conceptual design of a case-based mode management tutor, 
and the establishment of a Silicon Graphics research environment, and participation in the 
Training for Automation Workshop. Progress on each of these activities is discussed below. 

VNAV Tutor 

A statistical analysis of the data from the initial evaluation of the VNAV Tutor was completed. 
This evaluation used five pilots transitioning from non-FMS aircraft to a glass cockpit aircraft. 
The major findings from this analysis show that students learn the actions associated with VNAV 
well (see Figure 1). In most cases they successfully learn the relationship between the MCP and the 
FMS as it relates to VNAV (see Figure 2). However, the students are less proficient at identifying 
data sources to confirm the accuracy of their actions or the accuracy of the VNAV control mode 
operation. The data supporting these findings and more detailed interpretation are presented in 

Appendix Bl. 



A paper titled “VNAV Tutor: System Knowledge Training for Improving Pilots’ Mode 
Awareness” which documents the VNAV Tutor and the results of the evaluation was written. This 
paper was presented at the 1994 IEEE International Conference on Systems, Man, and Cybernetics 
and published in the conference proceedings. A copy of this paper and the presentation appear in 
this report as Appendices Bl and B2, respectively. 



In addition, we started the formulation of a plan for the follow-on evaluation of the VNAV 
Tutor. The VNAV related functions, data sources, and operations taught by the VNAV Tutor were 
identified and enumerated. This list extracts and summarizes the salient points of each tutor 
message in the four training scenarios. Such a set of training objectives defines the starting point 
for a thorough evaluation of the effectiveness of the VNAV tutor. Preliminary versions of the 
revised pre- and post- tutor questionnaires have been completed. The in-flight evaluation 
focusing on VNAV use and mode awareness has also been developed. These evaluation tools 
provide insights into the capabilities and limitations of the VNAV Tutor. The information gained 
through this evaluation will guide future Georgia Tech research in tr ainin g systems. 

Case-Based Mode Management Tutor 

In order to build on the lessons learned from the VNAV Tutor and other developments in 
training systems research we initiated the conceptual design for a new tutoring system, a case- 
based mode management tutor. This proposed system incorporates a case-base of flight scenarios 
(i.e., incidents and accidents drawn from the relevant literature, such as the ASRS data base). 
These scenarios are used to configure focused tutoring scenarios that highlight mode 
management, transition, and interaction. The tutoring system monitors pilot actions in order to 
control training both within and between scenarios. Within a scenario, the tutor uses a 
combination of expert and student models to guide the pilots focus of attention and flight control 
actions. Expert and student models also guide the selection of new scenarios to systematically 
broaden the pilot's exposure to mode operation and potential mode management problems. Such an 
instructional system, combining case-based reasoning (CBR) and intelligent tutoring system 
(ITS) technologies, may provide an environment and instructional content capable of teaching 
pilots the wide range of complex and varied modes of control in modern glass cockpit aircraft. 

Silicon Graphics Research Environment 

A Silicon Graphics Inc. (SGI) Indigo 2 was purchased and configured. This dual head 
graphics engine duplicates the Mini-ACFS environment at Ames Research Center. A comparable 
configuration promotes better software transfer from Georgia Tech to Ames and other Ames 
grantees. Future Georgia Tech aviation research will develop and evaluate proposed systems on 
the SGI . Experiments will be conducted with the research software running on the SGI 
communicating via UNIX interprocess protocols with the GT-EFIRT simulator running on a Sun 
SPARCstation 10. 



APPENDIX B1 


VNAV Tutor SMC Paper 


Crowther, E. G., Chappell, A. R., and Mitchell, C. M. (1994). VNAV Tutor: System knowledge 
training for improving pilots’ mode awareness. Proceedings of the 1994 International Conference 
on Systems, Man, and Cybernetics, San Antonio, TX. 



APPENDIX B2 


VNAV Tutor SMC Presentation 

Presentation Entitled “VNAV Tutor: System Knowledge Training for Improving Pilots’ Mode 
Awareness” presented at the 1994 International Conference on Systems, Man, and Cybernetics, 
San Antonio, TX, October, 1994. 



